9 research outputs found
LISSNAS: Locality-based Iterative Search Space Shrinkage for Neural Architecture Search
Search spaces hallmark the advancement of Neural Architecture Search (NAS).
Large and complex search spaces with versatile building operators and
structures provide more opportunities to brew promising architectures, yet pose
severe challenges on efficient exploration and exploitation. Subsequently,
several search space shrinkage methods optimize by selecting a single
sub-region that contains some well-performing networks. Small performance and
efficiency gains are observed with these methods but such techniques leave room
for significantly improved search performance and are ineffective at retaining
architectural diversity. We propose LISSNAS, an automated algorithm that
shrinks a large space into a diverse, small search space with SOTA search
performance. Our approach leverages locality, the relationship between
structural and performance similarity, to efficiently extract many pockets of
well-performing networks. We showcase our method on an array of search spaces
spanning various sizes and datasets. We accentuate the effectiveness of our
shrunk spaces when used in one-shot search by achieving the best Top-1 accuracy
in two different search spaces. Our method achieves a SOTA Top-1 accuracy of
77.6\% in ImageNet under mobile constraints, best-in-class Kendal-Tau,
architectural diversity, and search space size
PIDS: Joint Point Interaction-Dimension Search for 3D Point Cloud
The interaction and dimension of points are two important axes in designing
point operators to serve hierarchical 3D models. Yet, these two axes are
heterogeneous and challenging to fully explore. Existing works craft point
operator under a single axis and reuse the crafted operator in all parts of 3D
models. This overlooks the opportunity to better combine point interactions and
dimensions by exploiting varying geometry/density of 3D point clouds. In this
work, we establish PIDS, a novel paradigm to jointly explore point interactions
and point dimensions to serve semantic segmentation on point cloud data. We
establish a large search space to jointly consider versatile point interactions
and point dimensions. This supports point operators with various
geometry/density considerations. The enlarged search space with heterogeneous
search components calls for a better ranking of candidate models. To achieve
this, we improve the search space exploration by leveraging predictor-based
Neural Architecture Search (NAS), and enhance the quality of prediction by
assigning unique encoding to heterogeneous search components based on their
priors. We thoroughly evaluate the networks crafted by PIDS on two semantic
segmentation benchmarks, showing ~1% mIOU improvement on SemanticKITTI and
S3DIS over state-of-the-art 3D models.Comment: Proceedings of the IEEE/CVF Winter Conference on Applications of
Computer Vision. 2023: 1298-130
Farthest Greedy Path Sampling for Two-shot Recommender Search
Weight-sharing Neural Architecture Search (WS-NAS) provides an efficient
mechanism for developing end-to-end deep recommender models. However, in
complex search spaces, distinguishing between superior and inferior
architectures (or paths) is challenging. This challenge is compounded by the
limited coverage of the supernet and the co-adaptation of subnet weights, which
restricts the exploration and exploitation capabilities inherent to
weight-sharing mechanisms. To address these challenges, we introduce Farthest
Greedy Path Sampling (FGPS), a new path sampling strategy that balances path
quality and diversity. FGPS enhances path diversity to facilitate more
comprehensive supernet exploration, while emphasizing path quality to ensure
the effective identification and utilization of promising architectures. By
incorporating FGPS into a Two-shot NAS (TS-NAS) framework, we derive
high-performance architectures. Evaluations on three Click-Through Rate (CTR)
prediction benchmarks demonstrate that our approach consistently achieves
superior results, outperforming both manually designed and most NAS-based
models.Comment: 9 pages, 5 figure
AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
Resource is an important constraint when deploying Deep Neural Networks
(DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based
search approach, which limits the flexibility of network patterns in learned
cell structures. Moreover, due to the topology-agnostic nature of existing
works, including both cell-based and node-based approaches, the search process
is time consuming and the performance of found architecture may be sub-optimal.
To address these problems, we propose AutoShrink, a topology-aware Neural
Architecture Search(NAS) for searching efficient building blocks of neural
architectures. Our method is node-based and thus can learn flexible network
patterns in cell structures within a topological search space. Directed Acyclic
Graphs (DAGs) are used to abstract DNN architectures and progressively optimize
the cell structure through edge shrinking. As the search space intrinsically
reduces as the edges are progressively shrunk, AutoShrink explores more
flexible search space with even less search time. We evaluate AutoShrink on
image classification and language tasks by crafting ShrinkCNN and ShrinkRNN
models. ShrinkCNN is able to achieve up to 48% parameter reduction and save 34%
Multiply-Accumulates (MACs) on ImageNet-1K with comparable accuracy of
state-of-the-art (SOTA) models. Specifically, both ShrinkCNN and ShrinkRNN are
crafted within 1.5 GPU hours, which is 7.2x and 6.7x faster than the crafting
time of SOTA CNN and RNN models, respectively
Towards Collaborative Intelligence: Routability Estimation based on Decentralized Private Data
Applying machine learning (ML) in design flow is a popular trend in EDA with
various applications from design quality predictions to optimizations. Despite
its promise, which has been demonstrated in both academic researches and
industrial tools, its effectiveness largely hinges on the availability of a
large amount of high-quality training data. In reality, EDA developers have
very limited access to the latest design data, which is owned by design
companies and mostly confidential. Although one can commission ML model
training to a design company, the data of a single company might be still
inadequate or biased, especially for small companies. Such data availability
problem is becoming the limiting constraint on future growth of ML for chip
design. In this work, we propose an Federated-Learning based approach for
well-studied ML applications in EDA. Our approach allows an ML model to be
collaboratively trained with data from multiple clients but without explicit
access to the data for respecting their data privacy. To further strengthen the
results, we co-design a customized ML model FLNet and its personalization under
the decentralized training scenario. Experiments on a comprehensive dataset
show that collaborative training improves accuracy by 11% compared with
individual local models, and our customized model FLNet significantly
outperforms the best of previous routability estimators in this collaborative
training flow.Comment: 6 pages, 2 figures, 5 tables, accepted by DAC'2
NASGEM: Neural Architecture Search via Graph Embedding Method
Neural Architecture Search (NAS) automates and prospers the design of neural
networks. Estimator-based NAS has been proposed recently to model the
relationship between architectures and their performance to enable scalable and
flexible search. However, existing estimator-based methods encode the
architecture into a latent space without considering graph similarity. Ignoring
graph similarity in node-based search space may induce a large inconsistency
between similar graphs and their distance in the continuous encoding space,
leading to inaccurate encoding representation and/or reduced representation
capacity that can yield sub-optimal search results. To preserve graph
correlation information in encoding, we propose NASGEM which stands for Neural
Architecture Search via Graph Embedding Method. NASGEM is driven by a novel
graph embedding method equipped with similarity measures to capture the graph
topology information. By precisely estimating the graph distance and using an
auxiliary Weisfeiler-Lehman kernel to guide the encoding, NASGEM can utilize
additional structural information to get more accurate graph representation to
improve the search efficiency. GEMNet, a set of networks discovered by NASGEM,
consistently outperforms networks crafted by existing search methods in
classification tasks, i.e., with 0.4%-3.6% higher accuracy while having 11%-
21% fewer Multiply-Accumulates. We further transfer GEMNet for COCO object
detection. In both one-stage and twostage detectors, our GEMNet surpasses its
manually-crafted and automatically-searched counterparts
NASRec: Weight Sharing Neural Architecture Search for Recommender Systems
The rise of deep neural networks provides an important driver in optimizing
recommender systems. However, the success of recommender systems lies in
delicate architecture fabrication, and thus calls for Neural Architecture
Search (NAS) to further improve its modeling. We propose NASRec, a paradigm
that trains a single supernet and efficiently produces abundant
models/sub-architectures by weight sharing. To overcome the data multi-modality
and architecture heterogeneity challenges in recommendation domain, NASRec
establishes a large supernet (i.e., search space) to search the full
architectures, with the supernet incorporating versatile operator choices and
dense connectivity minimizing human prior for flexibility. The scale and
heterogeneity in NASRec impose challenges in search, such as training
inefficiency, operator-imbalance, and degraded rank correlation. We tackle
these challenges by proposing single-operator any-connection sampling,
operator-balancing interaction modules, and post-training fine-tuning. Our
results on three Click-Through Rates (CTR) prediction benchmarks show that
NASRec can outperform both manually designed models and existing NAS methods,
achieving state-of-the-art performance